1. Field of the Invention
The present invention relates to a signal processing apparatus such as a neural computer for modeling a neural network applied to character or graphics recognition, motion control such as a robot, and associative storage.
2. Description of the Invention
A so-called neural network is configured to have "neuron model cells" (neuron units) each of which models a function of a neuron serving as a fundamental unit for processing information of an organic body and is intended to process two or more pieces of information in parallel. Of information processings such as character recognition, associative storage, and motion control an organic body has a capability of simply achieving, a conventional Neumann type computer has difficulty in achieving many of such information processings.
To achieve those information processings, a great remark has been placed on the study about a model of a neural system of an organic body, in particular, a function peculiar to the organic body, that is, parallel processing or self-learning, in the field of computer simulation.
FIG. 1 is a model view showing one neuron model. This neuron model consists of a part for receiving an input from another neuron, a part for converting the input based on the predetermined rules, and a part for outputting the converted result. At the coupled portion with another neuron, there is put a variable weight "Tij" referred to as a coupling coefficient. The weight stands for a strength of the coupling. Learning means change of the weight value, on which the structure of the network is changed.
FIG. 2 is a model view illustrating a hierarchical network configured of several neural models shown in FIG. 1. This network consists of an input layer A1, a middle layer A2 and an output layer A3. One neuron is coupled with many neurons. In each layer, however, no neurons included therein are coupled with one another. Further, no coupling is provided from the output layer A3 to the input layer A2.
FIG. 2 shows a three-layered network having just one middle layer. In actual, a multilayered network may have two or more middle layers.
Considering a forward process executed in the network shown in FIG. 2. A signal inputted to the input layer A1 is propagated to the middle layer which the forward process is executed. The processed result of the middle layer is propagated the output layer in which the forward process executed. Then, the output layer feeds the final output of the neural network.
The degree of a coupling between neurons, that is, how strongly the neurons are coupled is represented by a coupling coefficient. In general, the degree of a coupling between the i-th neuron and the j-th neuron is represented by Tij. The coupling may be separated into two types. One type is a stimulant coupling in which as a neuron receives a larger signal, the neuron feeds a larger output. The other type is a suppress ire coupling in which as a neuron receives a larger signal, the neuron feeds a smaller output. Tij&gt;0 means the stimulant coupling and Tij&lt;0 means the suppressive coupling.
Assuming that the subject neuron is an i-th neuron and the j-th neuron feeds an output yj, the input to the subject neuron is Tij.multidot.yj in which Tij denotes the coupling coefficient between the i-th neuron and the j-th neuron(Refer to FIG. 1). Since each neuron is coupled to a lot of neurons, the sum of Tij yj for those coupled neurons, .SIGMA.Tij.multidot.yj, means an input to the subject neuron. This is referred to as an internal potential, which is represented by the following expression 1
Expression 1 EQU ui=.SIGMA.Tij.multidot.yi
Next, the nonlinear processing is carried out about this input, and the processed input is used as an output for the subject neuron unit. The nonlinear function used herein is referred to as a neuron response function. For example, a sigmoid function f(ui) is used as indicated in the following expression 2 and FIG. 3.
Expression 2 EQU f(ui)=1/(1+e.sup.-1)
From the above expressions, the yi-degree output of the neuron i is indicated in the expression 3.
Expression 3 EQU yi=fi(ui) =fi(.SIGMA.Tij.multidot.yi)
Later, as a learning process, a general back-propagation algorithm (abbreviated as BP algorithm) will be briefly discussed.
In the learning process, given a certain input pattern p, the coupling coefficient is varied as the error between an actual output vale and a desired output value is made smaller. The BP algorithm is an algorithm for deriving this variation.
When a certain input pattern P is given, a difference between the actual output value (ypi) of a neural unit i and a desired output (tpi) is defined as indicated in the expression 4.
Expression 4 EQU Ep=(tpi-ypi).sup.2 /2
This stands for an error of the output layer unit i. tpi denotes teacher data given by a user. In the learning process, the degrees of all the couplings are changed in a manner to reduce this error. In actual, the variation of the coupling coefficient Tij when a pattern p is given is changed according no the following expression 5. By using the expression 4, the resulting expression becomes the expression 6.
Expression 5 EQU .DELTA.pTij.varies.-.differential.E/.notident.tij
Expression 6 EQU .DELTA.pTij=.eta..delta.piypj
ypj denotes an input value from the unit j to
unit i. The error dpi is variable depending on
the unit i is the output layer A3 or the middle layer A2. The error signal dpi in the output layer A3 is represented by the expression 7 and the error signal dpi in the middle layer A2 is represented by the expression 8. In these expression 7 and 8, fi' (ui) is a result of differentiating fi(ui).
Expression 7 EQU .delta.pi=(tpi-ypi).multidot.fi'(ui)
Expression 8 EQU .delta.pt=fi'(ui).multidot..SIGMA..delta.pkTki
Based on the above-mentioned expressions, the general formulation of .DELTA.Tij is the following expression 9, from which the expression 10 is derived.
Expression 9 EQU .DELTA.Tij(n+1)=.eta..DELTA.piypj+.alpha..DELTA.Tij(n)
Expression 10 EQU Tij(n+1)=Tij(n)+.DELTA.Tij(n+1)
The first term in the right side of the expression 9 is a variation .DELTA.T of the coupling coefficient derived by the expression 6. The second term is added to reduce the vibration of the error and speed up the convergence.
As described above, the calculation of the variation .DELTA.DT of the coupling coefficient is started from the unit of the output layer A3 to the unit of the middle layer A2. It means that the learning is progressed conversely, that is, back propagated. Hence, the learning based on the back propagation is processed by repeating a forward process of inputting learning data and outputting the processed data and a back process of changing the degrees of all the couplings for reducing an error contained in the result until the convergence is completed.
FIG. 4 is a block diagram showing a general circuit for performing the forward process of one neuron based on the operation of numerical values. 2 denotes a memory for storing a coupling coefficient of a synapse and 3 denotes a multiplier. 4 denotes an area including the memory 2 and the multiplier 3. The area 4 serves as a synapse of the neuron for operating a product of an input signal and a synapse coefficient. 8 denotes a cell which includes an adder 5, a register 6, and a conversion table 7. The adder 5 serves to accumulatively add a product of a coupling coefficient calculated by the synapse and an input signal. This accumulated value is processed according to the sigmoid function contained in the conversion table 7. The processed value is output from the cell 8. This cell 8 holds true to the cell of the neuron in an organic body.
This is the content of the forward process executed in one neuron.
The conventional hierarchical neural network configures the network shown in FIG. 2. The flow of data in the forward process in the three-layered type network. As shown in FIG. 5, input signals a1 to a4 are given onto the input layer A1 and output signals b1 to b4 are given from the output layer A4.
FIG. 7 shows a multilayered neural network providing an input layer A, a plurality of middle layers B1 to Bn-1, and an output layer C. In the multilayered network, data flows from the input layer to the output layer through the middle layers in sequence.
Next, the forward process of the three-layered network will be described on a circuit diagram of FIG. 9 and a timing chart of FIG. 10. Herein, the circuit diagram of FIG. 9 indicates a circuit for configuring the three-layered network of the circuits for executing the forward process of one neuron shown in FIG. 4 and executing the forward process in the three-layered network. FIG. 10 is a timing chart showing the forward process.
At a state S1 of FIG. 10, an input signal 10 of the input layer is sent to a forward process executing circuit 11 located in the middle layer and a control circuit 15 starts to send a forward process start signal FWDS1 for the middle layer. With this start signal, the forward process is executed by the first input signal and the processed result is held in a register 12. At a next stage S2, a value held in the register 12 is inputted to a forward process executing circuit located in the output layer and the control circuit 15 operates to send a forward process start signal FWDS2 for an output layer. With this signal FWDS2, the forward process is executed in the output layer for obtaining an output 14. As described above, the sequentially given input signal 10 is forward-processed in parallel as shown in FIG. 10.
the multilayered network having a lot of middle layers, as shown in the circuit diagram of FIG. 11 and the timing chart of FIG. 12, like the foregoing three-layered network, the forward process is executed in a pipeline manner.
In addition, the learning process is basically executed in a general-purpose computer located outside.
FIG. 38 is a circuit diagram showing an implementation of the above-mentioned network on an electric circuit (see Japanese Patent Lying Open No. (KOKAI) 62-295188). Basically, this electric circuit is arranged to have a plurality of amplifiers 103 having a sigmoid transfer function and a resistive feedback circuit network 101 connecting an output of each amplifier 103 with an input of the amplifier on another layer as indicated by an alternate long and short dash line. On the input side of each amplifier 103, a CR time constant circuit 102 made of a grounded capacitor C and a grounded resistor R is connected. And, the input current I1, I2 to IN is fed to the input of each amplifier 103, the output of which can be obtained from the collection of output voltages of these amplifiers 103.
The strength of an input signal or an output signal is represented by voltage. The degree of a coupling of a neural cell is represented by resistance of a resistor 100 (lattices in the resistive feedback circuit network 101) connecting the I/O lines between the cells. The neural cell responsive coefficient is represented by the transfer function of each amplifier 103. Further, the coupling between the neural cells is separated into two types, stimulant and suppressive couplings as described above. Mathematically, it is represented by the positive or negative sign of the coupling coefficient. However, it is quite difficult to stand for the positive or negative sign by the constant on the circuit. In this circuit implementation, the output of the amplifier 103 is divided into two, one of which is reversed for generating positive and negative signals 103a and 103b. By properly selecting one of these signals, the positive or negative sign is represented. The sigmoid function as shown in FIG. 6 corresponds to the amplifier of this circuit arrangement.
FIGS. 33 to 36 are diagrams showing the neural network implemented by a digital circuit. FIG. 34 is a diagram showing a circuit arrangement of a single neural cell, in which 110 denotes a synapse circuit, 111 denotes a tree projection circuit, and 112 denotes a cell circuit. FIG. 35 shows an arrangement of the synapse circuit 110 shown in FIG. 34. A rate multiplier 110b is provided for inputting an input pulse f multiplied by a magnification a (which is a value to be multiplied by a feedback signal and is actually 1 or 2) through a coefficient circuit 110a. The rate multiplier 110 is connected to the synapse weight register 110c for storing a weight value w. Further, FIG. 36 shows an arrangement of the cell circuit 112 which is arranged to have a control circuit 113, an up/down counter 114, a rate multiplier 115, and a gate 116 connected in sequence. Further, an up/down memory 117 is provided to the cell circuit.
The I/O of the neuron unit is represented by a pulse train and the amount of the signal is represented by a pulse density. The coupling coefficient is treated by a binary digit and is held in the synapse weight register 110c. The signal processing is carried out as follows.
At first, an input signal is inputted to the rate multiplier 110b and the coupling coefficient is inputted to the rate value. With this, the pulse density of the input signal may be reduced according to a rate value. This corresponds to a part of Tijyi indicated in the expression 1 about the BP model. The part of .SIGMA. of .SIGMA.Tijyi is implemented by an OR circuit indicated by the tree projection circuit 111. Since the coupling has two types, stimulant and suppressive, the couplings are pre-grouped into two types, in each of which the coupling coefficients are ORed. In FIG. 34, F1 is a stimulant output and F2 is a suppressive output.
These two outputs are inputted to the up side and the down side of the counter 114 shown in FIG. 36 and the counter 114 counts the signals. The output of the counter 114 is a binary digit. Hence, again, it is converted into a pulse density by means of the rate multiplier 115. The plurality of neuron cell units are used for configuring the neural network.
The learning function is implemented by inputting the final output of the network to the external computer, doing numeral calculations inside of the computer, and writing the calculated result in the synapse weight register 110c for holding the coupling coefficient.
The present applicants have already developed a signal processing apparatus based on the neural network configured of neuron model cells and filed an application for a patent (Japanese Patent Application No. 1-343891). This invention concerns with this signal processing apparatus as a topic of one embodiment. Later, the description will be oriented to the signal processing apparatus developed by the present applicants.
this signal processing apparatus, as an example of the neural network, a proposal has been made about a neuron unit using a digital logic circuit and signal processing of a network circuit configured of such neuron units.
The basic ideas for this proposal are:
1. The I/O signal, the middle signal, the coupling coefficient, and the teacher signal about a neuron unit are all represented by a pulse train consisting of two values of "0" and "1".
2. The value of the signal in the network is represented by a pulse density (the number of "1's" within a predetermined time)
3. The calculation inside of the neuron unit is performed by logic operations about the pulse trains.
4. The pulse train about the coupling coefficient is stored in a memory located in the neuron unit.
5. In the learning process, when an error is derived on the pulse train corresponding to a given teacher signal and the coupling coefficient is changed on this error, the calculation about the error and the variation of the coupling coefficient is performed by logic operations about a pulse train consisting of "0" and "1".
Those basic ideas will be discussed below in detail.
FIG. 13 shows how one neuron element performs the forward process in a pulse density system. The configured network is a hierarchical neuron network as shown in FIG. 2.
At first, all the inputs and outputs of the neuron network are converted into a binary data consisting of "0" and "1". An input yi and a coupling coefficient Tij represented by a pulse density are ANDed for each synapse. This corresponds to Tij.multidot.yj in the expression 1. From a view of probability, the pulse density output from this AND circuit is equal to a product of pulse densities of an input signal and a coupling coefficient.
As mentioned above, the coupling of neurons are separated into two types, that is, stimulant couplings and suppressive couplings. For operating numerical values, the sign of the coupling coefficient is set so that the plus sign is for the stimulant couplings and the minus sign for the suppressive couplings.
For the pulse density system, each coupling separated into two groups, stimulant couplings and suppressive couplings based on the sign of the coupling coefficient Tij. For each group, the coupling coefficients are ORed. This logical OR operation corresponds to the .SIGMA. process of the expression 3 and the process of the nonlinear type saturation function fi.
That is, in the operation of the pulse density, if the pulse density is low, the ORed pulse density may approximate to the sum of the pulse densities of the ORed inputs. As the pulse density is made higher, the output of an OR circuit is made gradually saturated. The output of the OR circuit does not match to the sum of the pulse densities. The relation between them is nonlinear.
For this OR operation, the value P of the pulse density is 0.ltoreq.P.ltoreq.1 and has a relation of a monotonic function with the magnitude of the input. This is the same processing as that of the sigmoid function indicated the expression 2 or FIG. 3.
In the pulse density system, the neuron cell serves to output a value of "1" only if the OR output a of the stimulant group derived as above is "1" and the OR output b of the suppressive group is "0". That is, the outputs may be represented as follows.
Expression 11 EQU a=U(yi.andgate.Tij) (0&lt;T=stimulant)
Expression 12 EQU b=U(yi.andgate.Tij) (0&gt;T=suppressive)
Expression 13 EQU yi=g(a, b)=a.andgate.*b
Later, the description will be oriented to how the learning process is performed in the pulse density system.
In the neural network where no learning process is executed, the output of the network when a pattern is inputted is not always a desirous output. Hence, like the foregoing back-propagation algorithm, the learning process is executed to change the coupling coefficient toward reduction of an error between an actual output value and a desired output value.
(Signal Error in Output Layer)
At first, the error signal in an output layer will be discussed.
If the error is represented by a numerical value, it may take both positive and negative values. The pulse density system disallows such a representation. Hence, by using two signals standing for a plus component and a minus component, respectively, the error in the output layer may be defined as follows and illustrated in FIGS. 38-41.
That is, the plus component (.delta.+) of the error signal is "1" only if the output result is "0" and the teacher signal is "1".
On the other hand, the minus component (.delta.-) of the error signal is "1" only if the output result is "1" and the teacher signal is "0".
These error signals .delta.+ and .delta.- correspond to the expression 7 for obtaining the error signal of the output layer in the foregoing back-propagation algorithm.
(Error Signal in Middle Layer)
The error signal in a middle layer is also derived by referring to the expression 8 based on the BP algorithm in the pulse density system. That
the collection of the error signals in the output layer is defined as a self error signal. Herein, the coupling is divided into two groups, stimulant and suppressive coupling groups, in which the product is represented by .andgate.(AND) and the sum (.SIGMA.) is represented by U(OR). Further, to derive the error signal in the middle layer, tile coupling coefficient Tij has two types,
and minus and the error signal .delta. also has two types, plus and minus. In total, the four components are used for deriving the error signal.
For the stimulant coupling, the error signal plus component .delta.+ and its coupling coefficient of the output layer are ANDed. The ANDed result (.delta.+i.andgate.T+ij) is derived about all the neurons in the output layer. Then, those ANDed results are ORed. The ORed result is a plus component .delta.+ of the error signal of the neurons in the middle layer.
Expression 16 EQU .delta.+=U(.delta.+i.andgate.T+ij)
Likewise, the error minus component .delta.- and its coupling coefficient are ANDed. The ANDed result (.delta.-i.andgate.Tij) is derived about all the neurons in the output layer. Then, those ANDed results are ORed. The ORed result is a minus component .delta.- of the error signal of the neurons in the middle layer.
Expression 17 EQU .delta.-=U(.delta.-i.andgate.T+ij)
Next, the description will be oriented to the suppressive coupling. The minus component .delta.- and its coupling coefficient in the output layer are ANDed. The ANDed result (.delta.-i.andgate.T-ij) is derived about all the neurons in the output layers. Then, those ANDed results are ORed. The ORed result is a plus component of the error signal of the neurons in the middle layer.
Expression 18 EQU .delta.+=U(.delta.-i.andgate.T-ij)
Likewise, the error signal plus component and its coupling coefficient in the output layer are ANDed. The ANDed result (.delta.+i.andgate.T-ij) is derived about all the neurons in the output layer. Those ANDed results are ORed. The ORed result is a minus component of the error signal of the neurons in the middle layer.
Expression 19 EQU .delta.-=U(.delta.+i.andgate.T-ij)
The coupling between a neuron in the middle layer and a neuron in the output layer is stimulant or suppressive. As for the plus component of the error signal in the middle layer, the stimulant coupling .delta.+ indicated in the expression 16 and the suppressive coupling .delta.+ indicated in the expression 18 are ORed. Likewise, as for the minus component of the error signal in the middle layer, the stimulant coupling .delta.- indicated in the expression 17 and the suppressive coupling .delta.- indicated in the expression 19 are ORed. That is, the ORed results are the following expressions 20 and 21, which correspond to the expression 8 in the BP algorithm.
Expression 20 EQU .delta.+={U(.delta.+i.andgate.T+ij)}U {U(.delta.-i.andgate.T-ij)}
Expression 21 EQU .delta.-={U(.delta.-i.andgate.T+ij)}U{U(.delta.+i.andgate.T-ij)}
(Processing of a learning constant .eta.
The description will be oriented to the processing of a learning constant .eta. indicated in the expression 6 for deriving a modification ?T of the coupling coefficient in the BP algorithm and is seen in FIGS. 42-45. For the numerical operation, as indicated in the expression 6, the learning constant .eta. is simply multiplied with a multiplicand. For the pulse density system, according to the value of the learning constant .eta., the pulses are thinned out from the pulse train as indicated below.
Next, the description will be oriented to a method for deriving a modification ?T about the coupling coefficient in the learning process.
At first, the processing of the learning constant h is performed about the error signals (.delta.+, .delta.-) in the output layer or the middle layer. Then, the result is ANDed with an input signal to the neuron (.delta..andgate.y). Since the error signal has two types of .delta.+ and .delta.-, the operations indicated in the expressions 25 and 26 are performed about these error signals for deriving T+ and T-. These expressions correspond to the expression 6 for deriving ?T in the BP algorithm.
Expression 25 EQU .DELTA.T+=.delta.+.andgate.y
Expression 26 EQU .DELTA.T-=.delta.-.andgate.y
Based on these modifications T+ and T-, the net coupling coefficient New Tij is derived. In this case, the coupling coefficient Tij has to be separated into stimulant and suppressive.
For the stimulant coupling coefficient, the component of .DELTA.T+ is reduced as compared with the original T+ as seen in FIG. 46.
Expression 27 EQU New.sub.-- tij+=Tij+U.DELTA.T+.andgate..DELTA.*T-
Next, for the suppressive coupling coefficient, the component of .DELTA.T+ is reduced as compared with the original T- so that the component of .DELTA.T- may be increased as seen in FIG. 47.
Expression 28 EQU New.sub.-- Tij-=Tij-U.DELTA.T-.andgate.*.DELTA.T+
This is a learning algorithm in the pulse density system.
In the hierarchical network shown in FIG. 2, the executing flows of the forward process and the learning process in the pulse density system will be described briefly.
At first, in the forward process, when an input signal is given to the input layer, this input signal is propagated through the middle layer. As the signal processing for the middle layer, the foregoing expressions 11 to 18 are performed. The processed result is propagated to the output layer.
In the output layer, these propagated signals are processed according to the expressions 11 to 13. The process result is made to be an output signal. Then, the forward process is terminated.
In the learning process, after executing the forward process, a teacher signal is given to the output layer. In the output layer, an error signal is derived by the expressions 14 and 19 and then is sent to the middle layer. At a time, the processing of the learning constant .eta. is executed about this error signal according to the expressions 22 to 24. The processed result is ANDed with an input signal from the middle layer according to the expressions 25 and 26. Then, the coupling degree between the output layer and the middle layer is changed according to the expressions 27 and 28.
Next, in the middle layer, based on the error signal sent from the output layer, an error is derived according to the expressions 20 and 21. About this error signal, the learning processing of the expressions 22 to 24 is done with the constant h. The coupling degree between the middle layer and the input layer is changed according to the expressions 25 and 26. Then, the learning process is terminated.
This learning process is repeated until the convergence is completed.
Next, with reference to FIGS. 14 to 16, the actual circuit arrangement based on the above-mentioned algorithm will be described. The configuration of the neural network used therein is the same as that shown in FIG. 2. FIG. 14 is a circuit diagram showing a part corresponding to a synapse of a neuron. FIG. 15 is a circuit diagram showing a part corresponding a neuron cell itself. FIG. 16 is a circuit diagram showing a part for deriving an error signal in the output layer from the output of the output layer and the teacher signal. The inclusion of these three circuits in the network as shown in FIG. 2 makes it possible to implement a digital neural network for doing self learning.
At first, the description will be oriented to the arrangement shown in FIG. 14. 20 denotes an input signal to the neuron unit. The coupling coefficient of the synapse is stored in a shift register 27. At a terminal 27A, data is picked out. At a terminal 27B, data is inputted. In place of the shift register, it is possible to use another component having the same function, for example, a combination of a RAM and an address controller.
The circuit 28 is a circuit for executing (yi.andgate.Tij) in the expression 12. It serves to AND the input signal with the coupling coefficient. The ANDed output has to be separated into two groups according to the stimulant and the suppressive couplings. It is more available to prepare the outputs 23 and 24 to their groups and select the proper output for the object coupling type. Hence, a bit representing if the coupling is stimulant or suppressive is stored in a memory 33. A switching gate circuit 32 serves to switch the signal based on the bit.
Further, as shown in FIG. 15, a gate circuit 34 is provided to have a plurality of OR gates, which correspond to a logical OR executed in the expressions 11 and 12 for processing each input. Also as shown in FIG. 15, another gate circuit 35 is provided to have an AND gate and an inverter. This gate circuit 35 serves to give an output only if the stimulant group as indicated in the expression 13 is "1" and the suppressive group is "0".
Next, the error signal will be described. FIG. 14 is a circuit diagram showing a circuit for generating an error signal in the output layer. This circuit is a logic circuit having a combination of an AND and an inverter, which corresponds to the expressions 14 and 15. That is, the output 38 from the output layer and the teacher signal 39 are used for generating error signals 40 and 41. The expressions 16 to 19 for deriving the error signals in the middle layer are carried out in the gate circuit 29 consisting of AND gates as indicated in FIG. 14. The gate circuit 29 serves to give outputs 21 and 22 according to a plus or a minus sign of the coupling.
As mentioned above, the error signal is defined on whether or not the coupling is stimulant or suppressive. The separation is required to be done. This separation is performed by a gate circuit 31 consisting of AND and OR gates according to the information indicating if the coupling is stimulant or suppressive, stored the memory 33 and the error signals + and - 25 and 26. Further, the expressions 20 and 21 for collecting the error signals are performed in the gate circuit 36 consisting of OR gates shown in FIG. 15. The expressions 22 to 24 corresponding to the learning rate are executed in a frequency dividing circuit 37 shown in FIG. 15.
At last, the description will be oriented to a part for computing a new coupling coefficient from the error signal. This is represented by the expressions 25 to 28, which are performed by a gate circuit 30 consisting of AND gates, inverters and OR gates. This gate circuit 30 has to be separated depending on if the coupling is stimulant or suppressive. This separation is performed by the gate circuit 31 shown in FIG. 14.
The aforementioned hierarchical neural network has a configuration as shown in FIG. 2. Now, consider a forward process, as shown in FIG. 5, for giving an input signal to the input layer (layer A1 in the left hand of FIG. 5) and obtaining an output signal from the output layer (layer A3 in the right hand of FIG. 5) and a learning process, as shown in FIG. 6, for giving an input signal to the input layer, giving a teacher signal to the output layer in that state, changing a coupling coefficient between the output layer and the middle layer, and further changing a coupling coefficient between the middle layer and the input layer.
At first, in the forward process, when the input signal is given to the input layer, this input signal is propagated through the middle layer. The middle layer performs the operations indicated in the expressions 1 and 2 as signal processing and propagates the processed result to the output layer. The output layer also performs the operations indicated in the expressions 1 and 2 with respect to the propagated signal and feeds the processed result as its output signal.
In the learning process, after finishing the forward process, the teacher signal is given to the output layer. The output layer serves to derive an error by operating the expression 5 and propagate this error through the middle layer and change the degree of coupling between a neuron unit of the output layer and that of the middle layer, that is, a coupling coefficient based on the expressions 9 and 10.
Next, the middle layer serves to derive an error in this layer based on the expression 7. Based on the derived error, the middle layer changes a degree of coupling (coupling coefficient) between a neuron unit in the middle layer and that in the input layer based on the expressions 9 and 10. This is the end of the learning process.
The forward process and the learning process needs complicated processing and a vast amount of numerical operations as described above, which takes a considerably long time. In particular, the learning process has a larger amount of data to be processed than the forward process. Further, the learning process needs not a single learning operation but hundreds or thousands of learning operations. Hence, the operating time required therefor is so long that a dedicated hardware for this operation may not give a sufficient operating capability to the actual operation in the learning process.
The hierarchical neural network model may take a three-layered configuration as shown in FIGS. 5 and 6 as well as a various number of middle layers as shown in FIGS. 7 and 8. The larger number of layers takes a longer time in doing the forward process and the learning process.
FIG. 37 shows a flow of a conventional signal
the three-layered neural network as shown in FIG. 2.
In FIG. 87, the FWD circuits 4 and 5 are circuits for executing the forward process in the middle layer A2 and that in the output layer A3 in FIG. 2, which circuits correspond to 28, 32 in FIG. 14 and 34, 35 in FIG. 15, respectively.
The learning circuits LRN 6 and 7 are circuits for executing the learning process in the middle layer A2 and that is the output layer A3 as shown in FIG. 2, which circuits correspond to 29, 30, 31 as shown in FIG. 14 and 36, 37 as shown in FIG. 15, respectively. The error generating circuit ERR 8 serves to derive an error signal in the output layer, which corresponds to the circuit shown in FIG. 16.
As referring to FIG. 37, one learning process will be described. At first, when an input signal 1 is given to the input signal, this signal is propagated through the middle layer. The FWD circuit 4 in the middle layer serves to execute the forward process in the middle layer.
Next, the processed result in the FWD circuit 4 is propagated to the FWD circuit 5 in the output layer, in which circuit the forward process in the output layer is executed. This is the completion of the forward process. In the learning process, a teacher signal 3 is further given to the output layer. Based on the teacher signal 3, the error generating circuit 8 serves to generate an error signal in the output layer. The error signal is propagated to the learning circuit LRN 7. Based on the error signal of the output layer and the input signal to the FWD circuit 5, the learning process in the output layer is executed. The generated error signal in the middle layer is propagated to the learning circuit LRN 6 in the middle layer. Based on the error signal in the middle layer and the input signal to the FWD circuit 4, the learning process in the middle layer is executed. This is the completion of the learning process.
As set forth above, in particular, in the learning process, the processing path is so long that provision of the dedicated circuit for speeding up the process may not enhance its processing speed sufficiently. This is a disadvantage. Further, as shown in FIGS. 7 and 8, more middle layers need more processing time. This is another disadvantage.
It is an object of the present invention to provide a signal processing apparatus in a hardwared neural network which enhances its processing speed.